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v ) Abstract ' 

The study examined the effects on student achievement of (a) goal 

* • * • * 

size and data-utilization rule and (b) measurement frequency. Subjects . • 

v/ere 20 special education resource teachers, each of .whom *sel ected four 

to six students to participate in the study. Teachers .were assigned 

randomly to. either a Short-Term Goal Measurement or a Long-Term Goal 

-\ • '. * «... • • <. 

Measurement treatment teachers then randomly assigned e'ach student xo 
'daily, weekly, pre.-post measurement , so that, each teacher had one or 
two stthlents in each measurement frequency cell. Students' «oral reading 

\ - ) r 

rate wasimeasured at Weeks 1, 7, and 12 o.n random samples (of isolated 

9 i * 

words comprised of kindergarten through third grade reading vocabulary 

^Harrisf! Jacobson, 1972).- At' Week 12, students' pral reading rate was 
measured an third grade passages from basal texts. Analyses of variance , 
► •revealed no treatment effect on any dependent measure,/. This finding 
contradicts previous, research and is explained^ a,t least in part, by the 
poor implementation of the treatments.' "Daily" measurement was imple- 
mented* on a three-day per week basis ] and teachers adhered inconsistently 
to prescribed data r utilization rules.. Implications for training 'teachers 
are discussed. 1 



♦ » 



■r- 



» . 'The Relations-flip between Student Achievement and Teacher , . . 

" i v 

Assessment of Short- or LoagrTerm Goals 

• A * 

Federal, law (Public Law. 94-1.42) mandates -that schools develop. • 

• • • . * *' • ' . " . 

individualized educational plans for 'handicapped?students that specify 

"instructional goals and procedures fo.r measuring 'progress toward those • 

goals. To demonstrate substantive as well .as procedural compliance with 

•this law, measurement must.be incorporated into the instructional' process , 

because ongoing measurement creates the data base'-with which student pro- 

grams'can be monitored and improved continuously (Deno & Mirkin', 1980; 

••'Jenkins i Deno, & Mirkin?* 1979). - , * 

♦ * 

In response to this need" for* ongoing measurement of student progress, 

models of repeated curriCLrluirr-basetivassessment have been -developed (Dena 

& Mirkin, 1977; Covitt, 1977 ; White 1& Haring, 1980). ki rented currip- 

ulum-based assessment, students are jneasured 'frequently in their curricula, } 
« « - "* 

1 student performance* data are graphed, and these data are A nterpreted by * ^ 
teachers to determine wTipn and what program changes are required to im- 
prove student progress toward goals. 

fEach curricultim-based assessment system incorporates the specifica- 
tion of -three elements: a measurement behavior,, a measurement methodology,- 
and specific data' interpretation prcoedures . Although research has demon- 
strated that repeated -curriculum-based assessment positively affects stu- 
dent progress (Haring & jCrUg,- 1975 ^Haring", Maddux, & Krug, 1972; Jenkins, 

,*Mayhali; Vesdika, & Jownsend," 1974; firkin, Deno, Tindal, &* Kuehr>lel**'l980' f 

at the present timi it remains" unclear-which^ elements of the system are 

* related* to -improved student progress . The purpose of the current ; investi- 
> i *. . - * 

' gatiori was to, explore the relationship between some of the dimensions of/ 
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repeated curritfulum-bised. measurement and student achievement. . * " 
. - . <~ ** ; 

*. The 1 primary- question posed^in this study was^ Does measurement 

' . * 0 * : >• 

on, and-teacher assessment of, sfiort-term or long-term student goals 
' *• 
result in improved student a^hiev^merft? Concerned. both with measure- 

meht methodology and data interpretation, .this question asks whether 

teachers are more successful in designing,,, effective educational programs 

(a) when they measure' students and i'nterpret data on short-term objec- 

iives wherein the measurement domain changes frequently, or (b) wf\en they 

4 

measure students and interpret data on long-term goals covering a larger 

'measurement domain that remains constant over time. Measurement^ 

short-term objectives sfmilar to the measurement systems 4 practiced by 

precision teachers (Wh4|p & Haring, 1980); measurement on long-term 

objectives is analogous 'to the performance measurement systems described, 

by Deno and Mir^kin (1 977). - 

Additionally, two ^secondary questions were posed in this study. 
♦ 

V 4 

The fi rst* attempted to corroborate prior research demonstrating that 
frequency of measurement is related to .student achievement (Mirkin et al. 
1980; Omelich & Covington, 1981). It asked: Is student progress ^ 

'greater when students are measured daily oc "weekly? The oth^r secondary 
, question indirectly addressed the logi stical .feasi bi 1 ity*' of 'repeated , 
curriculum-based assessment by documenting whether teachers adhere-to 
designjafed measurement schedules and whether they follow^designateg data" 
util ization rules . ' 

■ ■ ' • . • - • : 

■ ■ ' , . Method \* ■ ' , .■ * • 

Subjects ' ? 

Twenty special education resource teachers -from a midwes/tern metro- 

" " • i ■ : ' " 

C 7 ■ : ■ - 



• • 3 

poll tan area v61unteered>t'o serve as subjects in the study. These 
teachers (2' males and 18 females) had taoght school for an average of 
9.6 years (SD = £.9 years). Each teacher selected four'tQ si> students 
from his/her caseload, resulting in a student sample(of 88 boys and 
20 girls. The students' mean age was 10.3 years; their mean grade level 
' was 3.9. • ' 

* 

Materials * . ... 

' * ' Training manuals. Teacher training manuals for two experimental 

— : — 17 ' 

♦ 

♦ 

conditions were written expressly for the study. Each manual consisted of 
six chapters on th§ assumptions of systematic instruction and 

procedures for placing students, setting' 1 ong-term goals, measuring stu- 
dents, recording and graphing data, and changing student programs. The. 
materials were self-instructional and each chapter concluded With a 

^mastery test. Two chapters,- "Measuring Students" and "Recording and 
Graphing Data," differed for the two groups; these chapters reflected the 
experimental treatments ^descri bed below. • 

Word cards . Word cards organized in instructional units from each 
student's existing reading materials were prepared for the teachers to 

' employ as fla^jicards in the measurement procedure. 
dependent Measures * "* 

Dependent measures were: v (a) nine lists of words, randomly sampled 
-from the Harris-Jacobson (1972) kindergarten through third grade reading 
vocabulary (K-3 Lists)-; and (b) three third grade 'level reading passages. 

Procedure ' A . * 

Experimental conditions . -Teachers were asstgned randomly to one^ 
of two experimental' treatment groups for the purpose of measuring student 



progress: Long-Term Goal Measurement (LTGM) or Short-Term Goal Measure- 

ment (STGM). 'In the LTGM conation, teachers tested students 1 ,reading 

performance by administering a 30-second word recognition test comprised 

of 25 words that wer£ randomly selected from a large set of vocabulary 

Words to be introduced within the 12-week period. At each measurement 

4 session teachers graphed the -student' s performance; on the sixth through 

ninth days, they were required to review the graphed data. If these data- 

♦ 

indicated that^ progress was inadequate, then the v teachers were to intro- 
duce an adjustment to improve* the effectiveness of the- instructional*. pro- 
gram. After 10 days, teachers were required to make an adjustment if one 
had not been made previously. This routine was repeated several times in 

the 12-week experimental per.iod. - (See Figure 1 for exampje of a graph.) 

f 1 



Insert Figure 1 about here 



In the STGM group, ^teachers tested), a student's reading performance b 
by administering a 30-second word recognition test' comprised of 25 words 
that included all vocabulary words introduced in the current instructional 
period plus words sampled from preceding stories. Teachers graphed thfe 

student's performance and compared that performance against a, short-term 

v 

aim line that the teachers, drew on a graph each time a new shprt-term 
goal was established. Teachers were asked to review the data frequently 
and to determine" when to progress to the next sttfry and/or when to make 
a program adjustment. Th'is routine continued throughout the^ 12-wee-k 
period. (See Figure 2 for example of a graph.) \ 



Insert Figure 2 about here 



Irrespective of their assignment to either the long- or short- 
term <go a 1 measurement condition, the teachers randomly assigned their 

students to* one of three frequency-of-measurement condilfons: daily \ 

« 

measurement, weekly measurement, or pre-pbst measurement. This resulted 
in a ohe-between factpr (frequency-of-measurement condition) experimental* 
design with five cells: Daily Measurement of Loncf-Tern> Goal s , Wee.kly 
Measurement o^ Long-TeniLGoals , Daily Measurement of Short-Term GoaU, 
fieekly" Measurement of^ Short- T*erm Goals, and Pre-post Measurements 

Training . At the first of *two, 1 1/2 hour sessions, teachers were 
trained in procedur.es for placing studgnts^in curricula. These procedures 
required students to read, in the currently employed basal reading series, 
thcee one-minute samples on each of the three highest levels at which the 
'student could read approximately 30-49 words per minute (wpm) with few^r 
^than eight errors for poor readers, and 50-99 wpm with fewer than eight 
errors for 'better^readers . After three days of data collection, teachers 
were to* compute median wpm and median errors per minute for each /level, 
and place the student at the highest; level at which the instructional, 
criterion was met. 

Having been trained in this placement procedure, teachers implemented 
it wfien they returnedto their schools. Although teachers had been 
advised to place a student in the highest level'at which the stucfent met 
the performance standard, they often used other criteria s^ch as previous 
placement, intuition, and logistical feasibility of placements. Between, 
the tf r aTni?rg sessions, the teachers' also .compl eted reading and answering 
questions in the train.ing manuals. • 
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, At the second training session, tpai nincfmanual mastery tests* 

L • ■ • ' 

wewe scared. Additionally, as instructed in the, manual., teachers set 

long-term (12 week) goals for* progress , and submitted a Hst of all I 

vocabulary words contained in those 12-week goals; These words were 

made into word ,card packs. 

•* *, 
Teacher visi ts . One week after the second tr-aini ng session, a grad- 

uc^te research assistant (Redelivered the worti packs to each teacher, 

and helped the teachers set up student graphs. Teachers then began to 

implement the treatments: , * * 

' jOm RA was assigned to each teacher. During the 12-week treatment 

period, RAs made weekly 10- to 2,0-minyte v.isjfts to their assigned m teachers 

to.monitbc teacher activities and provide additional training as required. 

* * 
Administration of dependent measures . During Weeks 1, 7, and 12 of 
v ♦ 

.the study, teachers administered three K-3 Lists to the students, each 
list on a* different day of the weak. -On the Week 1 and 7 tests, .students" 

'*r£ad individually fpr 30 seconds; on the Week 12 test, students read - 

individually for, 60 seconds. Additionally, during Week 12, three 60- 

second oral reading passage tests were" admini stered~ individually to the 

Students each on a different "day. On all tests, students were. directed 

to read as quickly and accurately 'as they couTd and .to skip* unknown words. 

Only completely, accurate responses were scored as correct. 
> • 

Results 

Student Achievement • * 

Tables 1-3 present the means^and 0 standard deviations for students ' 
performance on the dependent measures. For every analysis, t\h dependent 
data were e'ach subject's correct and incorrect sccfres;* these were computed 
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by averaging across the three administrations .of measures each week. 



Insert Tables 1-3 about here 



One-between factor (LTGM vs. STGM-) and orie-withfn factor (daily vs. 

weekly vs. pre-post measurement) analyses of variance were run on the 

following s'tudent achievement data:- fa) Week 1 means on K-3 Lists, (b) 

Week 7 means on K-3 lists, (c) Week 12 means on K-3 lists, (d) Week 12 * 

e meew(s^on reading passage. Additionally, on the word recognition tests, ' 

one-between ^factor (LTGM vs. STGM) and two>within factor (daily vs. 

weekly'vs: pre-post; Week 7 vs. Week 12) analyses of variance were run, 

once with the ^Orsecond' K-3 List^gretes.t included as a covariate and once 

without the covariate. * ... % 

Analyses revealed statistically significant ^ins (£ = .001) on the 
< • * . I * c - 
average correct scores for th$* entire group from Week 7 to Week 12, both 



. with and without^the word recognition pretext included as* ar covariate , 



No.other statistically significant differences were found'. 
Treatment Implementation * ' ^. 

Foi> each "student, "investigators counted (a'fthe number of calendar 



days over which the study spanned,. and 4b) th* number of days on which 

« * . ^ ' ' f 

measurement occurred. These were averaged and compared (t tests) for the 
frequency of measurement conditions (dail.y vs. wefcMy) and for the . 
•measurement format conditions. (STGM & LTGM). No statistically signifi- 

cant differences were revealed for the number of calendar d^ays. ^However, *X 

* • * * " % 

for the number of measurement days, there was r a statistically significant, 

difference (j) = «.000) between the daily and weekly measurement groups, 
* # 

with approximately thrge times more measurement days in the daily group 

* * 

: 12 .-• ' ' ' ' 



(see Table 4).- 
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* Additional analyses addressed the precision with which teachers had 

implemented the data utilization rules. For STGM, a decision to" move 
♦ • ■ 

to a 'new word list, was optional each day. If a student's performance 
had met the, mastery criterion, then a correct decision was to progress 
to^a new word list; if the performance level was lower than the estab- 
lished criterion, then a correct decision was not to move ahead. The 
percentage of measurement days on which the teachers correctly determined 
whether to progress to the next word list was\ajculated. In the daily ■ 
'STGM condition, teachers moved appropriately to new "hi s ts an average of 
66% of measurement days. Approximately one-third of all errors made, by 
these teachers were incurred when teachers moved to new wpVd lists even 

V. * > 

though students had not met mastery criteria. Approximately two-thirds 
of errors »were* characterized by teachers failing to introduce a new word 
.list despite the fact "that students had reached performance standards. -~ 
For the weekly STGM condition, 6»of 12 gr^h^ were difficult to read, and 
statistics for this group are nat reported here due to the threat of un- 
reliable, nonrepreservtati ve results/ % 

For LTGM,. a decision tO{ introduce a new intervention Was optional , 
on the sixth through the ninth measurement days following, the introduction 
of a program change. If the student's performafice'accelerated from one 
day to the next, 'the appropriate deqisioji was to maintain the current pro- 
gram. If the student 1 s k pei^fprmarice was flat or decelerating, then the 
correct decision was to introduce a program change. The percentage o<* 

• 13 - 



days on v/hich LTGM^teacfiers correctly utilized this decision rule was 

calculated. Fifty-six percent of the dally LTGM decisions were correct; 

78% Qf weekly LTGM decisions were .correct. -*» * 

j 

J 

4 » 

9 • 

Discussion 

In this investigation ,* students made achievement gains within a 12- 

r 

week intervention, period; however, the size of those gajns was unrelated * 

to the measurement strategies employed. Students who received only pre- 

post measurement showed progress similar to that of students v "who were 

measured repeatedly either on shortPte$h or long-term goals. Additionally, 

students who were measured daily progressed at a rate similar to that of 

«# -. 

students who were measured weekly. With respect to questions posed in the 
present study, it appears that (a) measurement on and teacher assessment 
of short-term and long-term goals does not result in differential student 
gains, and -(b) student progress is not affected by the frequency of the 
measurement. 

These findings contradict those of earl ier 'studies , which have 
demonstrated a relationship between student achievement and da'ta utiliza- 
tion strategies (Martin, 1980; Mirkin & Deno, 1979; Mirkin et al., 1980) 
and a relationship between student achievement and frequency of measure- 
ment (Mirkin* et al., 1980; Omelich & Covington, 1981). Moreover, this 
study contradicts previous research establishing that repeated curriculum- 
based assessment -posi ti vely affects student progress (Haring & Krug, 1975; 
Haring et al., 1972; Jenkins et a.l . , 1974). , % 

The confrafdictions from the present study may be explained by the 
questionable treatment implementation documented in this study. Results 
revealed that the distinction between daily and weekly measurement was less 
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dramatic than had been intended: teachers implemented "daily" measure- * 
fc ment on a three day per week ba'sis. % This lack of adherence to the 
designated measurement schedule may explain the noted lack of relation- 
ship between measurement frequency .and student achievement. Because the 
teachers collected weekly data but failed to collect-daily data, this 
study suggests that such frequent measurement may be logistically diffir,. 

cult for teachers. In factj in follow-up interviews, teachers indicated 

4 \ 
that daily measurement was time consuming and difffcult. Such findings 

document the need for developing procedures that mfght improve' the feasi- 

bility of collecting freq'u§nt studeat data. 

Teachers in this study also implemented the designated decision 

rules imprecisely, with teachers in the daily STGM condition appropriately 

% - 4> 
moving to new lists on only two-third? of measurement days. Teachers 

in -the daily and weekly LTGM conditions correctly employed their decision 

rules on 56% and £8% of occasions, respectively. The relatively high 

percentage of correct judgments for the weekly LTGM g^up may be related 

' to .the few data points in weekly measurement and the mandated program 

change every 10 school d$ys which left relatively few opportunities for 

incorrect judgments. \ 

Therefore, ^though teachers collected student data in this invesl 

gation, it appears that they incorrectly, implemented the data-utilization 

. 

rules. Conceivably, they employed the' col leic'ted student data minimally, 

V «* 

if at all, to ^formulate decisions about student programming. If so, it is 

not surprising that the achievement gains of tfteir students in^he STGM 

and LTGM conditions failed, to surpass those of students vn the pre-post 

♦ 

measurement ^group.< • 

•is . 



Previous studies have documented that teachers who collect student 

— y ' 

performance data do not necessarily use those data to make instructional 
decisions (Baldwin, 1976-; White, 1974). The current investigation indi- 
cates, that data utilization procedures may be an essential dimension of 
a measurement system that .is effective in -improving student achievement. 
It further suggests that effectively training teachers to* interpret 
correctly and use consistently student data may be critical. 
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Table 1 



Means and Standard Deviations of Number Correct on 







3Q-Second "k-3 Lists at Weeks 1, 7, and 12 


a 


• t 

i 

"> 

^ ■' 








STGM • 








' LTGM' 






Wk 1 


Wk 7 


Wk 12 


/ Wk 1 




Wk 9 


" T 

Wk.12 


Daily 


14.1 (9.6) 


17.6 (9.9) 


43.2 (23.2) 


12.3 (5.6) 


15 


.9. 


> 

34.5. (9.8) 


Week-lj(' 


16.7 (9.0) 


18.0 (10.2) 


43.1 (23.3) 


19.2 (7.2)' 


21 


..0 (7.9) 


„,42.6 (17.4) 


Pre- Post • 


14.5 (13.3) , 


14.5 (14.3) 


48.2 (36.5)- 

- \ 


18.3 (7.5) ' 


19 /l ) 


42.5- (16.6) 



* T 

a Entries in table are means -and standard deviations (in parentheses) of the average number correct 
.across the three administrations of the test given during the week. 



Table 2 

Means and Standard Deviations tff Number*! ncorrect on - 

* a 
30-Se'cond K-3 Lists ^at Weeks 1, 7, and 12 



STGM 



LTGM 



Wk 1 



" Wk 7 



Wk 12 



Wk 1 



Wk 7 



Wk 12 



Daily 8.3 (4.Q) ' 7.5 (4.9) 18.8 (12.7) 

S ' * 

eekly 6.7 (3-8) 5.6 (4.2) 11.4 (9.6) 

Pre-Post ' 6.6 (5.8) 6\5 (4.9)A 9.9 (11.3) 



6.6 (2.8)- 6.9 (-3.1) 12.9 (7.9) . 

5.6 ' (3M) ■ 5.0 (3.1) 10.3 (7.3) 

* • * t . ' 

6.3 (3.7) 5<8 (3.7) 12.2 (8.1) 



Entries in table are means and standard de'viations (in parentheses) of the average number incorrect 
across the three administrations of the test given during the week. 
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Table 3 

• Means and Standard Deviations of 'Number Correct and* 

* % 

Number Incorrect on 60-Second, Oral Reading Test at Week 12- 





Number Correct' 




Number Incorrect 




STGM 


LTGM 

♦ 




STGM . ' • 


LTGM 


Daily 


80 .1 (24. §) 


69.1 (20.0) 


' 6 


.1. 


' 5.2/ [i.ti 


Weekly 


74.0 (26.6) 


79.4 (18.1) 


5 


.5 (3.4) 


4.6* (2.8)" 


Pre-Post 


83.2 (50.9) 


79.2 (21,3) 


5 


.6 -(4.7) 


'5.1 (2.7) 

1 '—, 



Entries in table are me^m^and standard deviations (in parentheses) 
of the, average number correct or ificorrect^ across *the three ad- 
ministrations -of the test during Week .12 . - ' . . • . 



Table 4 



Means and Standard Deviations 'of Numbers ^of 
Calendar and Measurement Days** -V; 



V 



Condition v 



, STGM • 



LTGM 



laily 



». Weekly 



Calendar Days 104.0 (8.1) 104.0 (5\l) 103.9 (fc.:8) 104;0 (6;7) 

f^asjurement Days '30.1(19.1) 31.0(18.6) - ' 4? £ J9 J) , + 4 ;3 (7'.2j . 

r • • ' '> v ' - "" i • 

a Entries are means and standard deviations (in parentheses bof th,e numbers of daysx. 
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Figure, 2. Example of graph for Short-Term Goal Measurement. condition* 

o \ * - 

to* Ml * ♦ 
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